The present invention relates to a system and method for providing sounds to be rendered in real-time to accompany a silent computing application.
Silent computing applications are those that do not produce sounds as their primary functions but which can become more informative when the production of sounds is added to their interfaces and displays. Many silent applications are interactive and involve real-time graphical displays. Computer animation applications, computer based games, Internet browsers, digital multimedia production applications and scientific visualization software are examples of silent applications. These applications may be implemented in hardware or software.
Several different techniques have been used to add sounds to or incorporate sounds with silent applications. These prior techniques may be characterized as falling into one of the two general categories of conventional approaches shown in FIGS. 1 and 2.
The block diagram of FIG. 1 represents one general category of existing, conventional, off-the-shelf, interactive audio technologies for enhancing computing applications by adding sounds. In FIG. 1, a silent application 10 includes a routine 12 that evaluates dynamics 16 of the silent application 10. In response to the evaluation of the program dynamics 16, the silent application 10 sends function calls 20 to libraries 124A and 124B that are supported by the particular hardware operating system 102. These libraries 124A and 124B can generate sound in several ways: One way that the libraries 124A can generate sounds is by accessing audio playback functionality of the system, identifying a digitized sound stored in the system, and playing the sound in a linear fashion from beginning to end. Another way these libraries 124A can generate sound is by sending a MIDI message to a peripheral sound-generating device 127, such as an internal synthesis card or an external synthesizer 128, and producing a MIDI note event. Still another way these systems can generate sound is represented by the dotted line in FIG. 1. In this latter approach, sound is generated by the libraries 124 by accessing sound synthesis functionality 89 of the system, such as sound synthesis languages and compilers 90, sending a request to the system accompanied by parameter values if the system accepts them, and synthesizing and playing a sound event. This type of sound synthesis functionality is not available on many platforms.
In the first two approaches described above in connection with FIG. 1, no sound sample computation occurs in the computing platform. The third approach described above (i.e., the dotted lines in FIG. 1) represents the hypothetical presence of a software synthesis language capable of running in real-time on the computing platform. Existing synthesis languages are not in wide use in multimedia computing due to their idiosyncrasies, their computational intensity, and their lack of an intuitive interface for software developers.
A fundamental drawback of the general category of approaches represented in FIG. 1 is that they all enforce and assume a one-to-one relationship between a request for a sound and a sound synthesis event (many classes of sound cannot be created from a single sound synthesis event). These approaches further assume that the requesting application 10 will encode knowledge of audio library functions 124 to initiate and control sounds. Therefore, the requesting application 10 must encode specialized knowledge about sound function calls and specialized sequences of function calls to control the sounds if they are not self-contained single events.
A generally more sophisticated category of approaches to adding sounds to silent applications is represented by the block diagram of FIG. 2. The approaches represented in FIG. 2 may be used presently in professional media. Audio production standards in professional media utilize many layers of sound to create the auditory impression of a real-world acoustic space populated by sound-making objects and events. Professional sound production standards require multiple-event instruction sets instantiating and modifying multiple sounds in parallel. In an interactive application, these instruction sets need to be generated by algorithm. The approaches in FIG. 2 represent the application of conventional, existing technology to implement an event list/algorithm scenario. Dotted lines represent optional locations for computing sound control algorithms in the rendering path.
In FIG. 2, the algorithms 130 may be encoded in the sound requesting application 10. Digital control signal output may be sent to synthesis directly (e.g. via path 135) or stored in a file as an event list 138, which is a time-ordered script of synthesis instructions. Algorithms may also be encoded in another application residing on the host platform or on an external platform communicating via a serial or SCSI connection. The MIDI processing language MAX (labeled "160" in FIG. 2) is an example. In the case of MAX, numerical data pass from the requesting application 10 and MIDI messages are generated from algorithmic processes. As in the category of approaches in FIG. 1, the sounds may be generated by MIDI libraries 124A or soundfiles 124B that access peripheral devices 127, such as synthesizers 128 or a MIDI processing module 166, to provide an audio output 129.
A disadvantage of the approaches depicted in FIG. 2 is that the location of specialized audio domain knowledge in the code of a silent application 10 is an impediment to software developers and a constraint on the flexibility of sound prototyping. In the category of approaches in FIG. 2, extensive interpretation for generating sounds needs to be encoded in the requesting application 10. This requires specialized knowledge, and it requires re-programming and recompiling the silent application 10 whenever one wishes to modify the rules for audio output.
A further disadvantage associated with the approaches of FIGS. 1 and 2 is that the conventional use of MIDI represents a significant reduction of control over the details of sound production. The MIDI protocol provides no encoding of specific synthesis instructions nor algorithms and places most of the dynamics of sound production beyond the control of a user. With the category of approaches depicted in FIG. 2, there is no memory location or language provided for describing a relationship between the dynamics of the silent application and the dynamics of sound production. Each dynamic resides in a separate memory location in a separate system defined in a separate language.
Accordingly, one object of the present invention is to provide a way to enhance a silent application by adding sound by means of a process that is flexible and easier-to-use than conventional methods. Another object of the present invention is to provide a software architecture that enables sophisticated audio enhancements to silent applications in a way that is configurable and expandable. Still another object is to describe the relationship between silent application dynamics and sound dynamics in a uniform language and to store that description in a uniform memory location.